12. Misc topics

Today's topics

Ubicomp

Helpfeel

Generative Art and design

History of UI

Final task

Deadline: 2020/8/8

Ubicomp today

Integration with Web

Cheap and practical

Current IoT situation

by Ken Sakamura

http://dol.ismcdn.jp/mwimgs/8/8/670m/img_881643ea16a0670f95774a6457dc6ee06718695.jpg http://diamond.jp/articles/-/130230

What you cannot find in Google

Lock status of my home

Good restaurants

People I met in Shinjuku

Lost smartphone

These can be found using UbiComp technologies

Ubicomp challenges

Integration with the Web

Hardward obstacles ⇒ disappearing

Software obstacles ⇒ disappearing

Integaration with the Web

http://gyazo.com/82174232cc357d0afb5b8e6b21afbc92.png

Comic on electronic DIY (2014)

Published on "Weekly Morning" comic magazine

https://gyazo.com/214e2fcc8510efad29b9a4c1f7753cc4

Natchan

Lathe, welding, ...

https://www.amazon.co.jp/dp/B01KXQQG30 https://gyazo.com/f84eadcb457154543340bf25df736c07

Homecenter Tenco

https://www.amazon.co.jp/dp/B00XBG717K https://gyazo.com/133ecddb94d9db8629990c2b9e162c77

Hardware obstacles disappearing

Phidgets, Arduino, mbed...

Phidgets

USB-connected sensors

http://gyazo.com/fef02a278bcd20cf8af4323fb2fbff8e.png

Arduino

USB/Serial + AVR

Programmable with Processing environment

http://gyazo.com/ba0f1f2dd89edcd0b5d9d2630fec4e04.png

Arduino Micro

http://gyazo.com/a49ee2e6b0ecb16faeaf45b9ae5d5b1d.png

Arduino + Bluetooth

http://gyazo.com/52298c7657576812dc4117be888b5909.png

LilyPad

Arduino for wearable computers

http://gyazo.com/d69ad7cad9a36bcd1e9cd4e315b7dd1a.png

LilyPad

http://gyazo.com/54cd28dda035ebee0eff4efc33ee8d51.png

mbed

Programmable on the Web

http://gyazo.com/d21d725a11af5e9cd6cdba2de538b7e8.png

Harpy

Android ADK+Arduino

http://gyazo.com/9c8d4389e07adbea93ced74f290417d7.png

Raspberry Pi

ARM-based one-board computer

Linux

https://gyazo.com/de22bebac62ffc048f3146ed5a55ff60

http://gyazo.com/8eec26e91d9b97c806de1e32c6cabbcf.png

Intel Galileo

Linux + Arduino

http://gyazo.com/022769af99c411731730a31e3bec67d0.png

Intel Edison

http://gyazo.com/990178c9f83a96d1d2d779f40af7dbd0.png

M5Stack

https://gyazo.com/21da7030b9f452cfacd9229be6773e38

320 x 240 TFT color LCD display

microSD

Speaker

WiFi

Bluetooth

Battery

USB-C

M5Stack

https://www.youtube.com/watch?v=OR4qwGzzHAk

M5Stack

https://www.youtube.com/watch?v=Yu9A5pYnjCo

M5Stick

https://www.switch-science.com/catalog/5517/ https://gyazo.com/8d728b596ea3ed08f64fa572c6a1ef7e

Smaller M5Stack

Software obstacles disappearing

Simple and powerful languages

JavaScript

Processing

Various languages and libraries available

Ubiquitous == Universal design

Anytime / Everywhere / Anybody

Enhance human abilities

e.g. glasses

Human enhancement

Human augmentation

Intuitive operation available

Basic technologies for intuitive IT systems

Small computers

Wireless network

Sensors

Interaction techniques

How to make intuitive interaction

Smooth interaction

Reversible / continuous

Direct connection between interaction and goal

e.g. Automatic door

UbiComp with abundant resources

Use many special sensors

Use many displays

Use special computing environment

Don't use special sensors

No camera, no 3D position sensor

No special display

No HMD

Use small number of robust off-the-shelf sensors

GPS phones, accelerometers

Requirements for PMCC

Basic infrastructure

Internet, WiFi, etc.

Interaction techniques

Input /output devices

Sensors and actuators

Requirements for PMCC

Infrastructure

Internet, WiFi, etc.

Information retrieval techniques

Input/output devices

Sensors and actuators

We already have all of them!

http://gyazo.com/2b51c7542fbc81b94d3d4a5719e1c1a4.png

Obvious problems

Too many buttons and difficult to understand meanings

Easily lost

Obvious problems

Too many buttons and difficult to understand meanings

Easily lost

Real problems

Controllers should pointo to machines that is not related to what users want

Not intuitive at all

Right approach

Do whatever you want without thinking about devices

Watch movies on the wall of a house

Listen to music in the bath or in the kitchen

Users don't worry about what device they use

Nagaya computing

Can you survive here?

http://gyazo.com/498cf3e4b9c769d7aa35299b5c7cf213.png

How about this?

http://gyazo.com/498cf3e4b9c769d7aa35299b5c7cf213.png

WiFi installed

Large display (on shoji)

Server somewhere

Ubiquitous keyboards for input

Air conditioners

Electric books

Nearby restaurants

Nearby convenience stores

Nearby spas

Non-intuitive operations

Switching on/off

Swithches are located around the entrance

Watching DVDs

Users have to control DVD player, projector, amplifier, screen, speaker

Difficult to handle all of them

Video recording

Users just want to watch programs

Window open/close

Intuitive CD player

Play music only by putting a CD on the table

https://www.youtube.com/watch?v=av7j9Juj6To

https://s3-ap-northeast-1.amazonaws.com/masui.org/5/a/5a3e607da880440d50fefb77b1f17a38.jpg

Intuitive data transfer

When CD is inserted, data is transferred to a remote machine and CD is burned

https://s3-ap-northeast-1.amazonaws.com/masui.org/3/9/3982e892796a6c7b140b2601e751de50.jpg

https://s3-ap-northeast-1.amazonaws.com/masui.org/f/3/f3e59ee8d9285932cc8b9e43b5f0652f.jpg

Cash register for handicapped

No need to understand coins and numbers

https://s3-ap-northeast-1.amazonaws.com/masui.org/e/8/e8b7b8735831648fa033cf7633ca5c38.gif

https://s3-ap-northeast-1.amazonaws.com/masui.org/3/2/326b8a7a35769f3f55510ed6296c4286.gif

Strange automatic door

https://s3-ap-northeast-1.amazonaws.com/masui.org/2/c/2c591cc84eb7a1827df6bf4bc5f38365.jpg

https://s3-ap-northeast-1.amazonaws.com/masui.org/d/6/d68556ddbac9971d7809338ebbf611b5.jpg

Input devices for PMUC (Poor Man's UbiComp)

Cheap and robust

Anybody can use

Look familiar

Requirements

Simple operations

Usable in harsh environments (bath, kitchen, etc.)

Restricted interaction (one-hand, sound, etc.)

Small display

Real-world GUI

Use GUI in the real world

Buttons, menus, sliders, drag&drop on the desk and wall in the real world

Use FieldMouse, MouseField

Real-world copy/paste

https://www.youtube.com/watch?v=1di7_2J8X0Y

FieldMouse

ID recognizer + motion recognizer

Use it as a mouse on a paper or on a wall

Active picturebook

Paper-based remote controller

Real-World GUI

https://s3-ap-northeast-1.amazonaws.com/masui.org/7/3/73bb7ceed3106ba563c784a232fbf107.jpg

https://s3-ap-northeast-1.amazonaws.com/masui.org/4/f/4f9a8573e720ba8c0f9b18c5d2e7ce45.jpg

Active picture book

Use a paper book as a tablet

https://www.youtube.com/watch?v=kuZW-znOGIA

https://s3-ap-northeast-1.amazonaws.com/masui.org/f/a/fa094f15e1e33761101f9933448622c6.jpg

Paper remote controller

Use a paper as a remote controller

https://s3-ap-northeast-1.amazonaws.com/masui.org/4/5/4598bb1047ffa112b3a4594536dd202e.gif

FieldMouseを使った実世界GUI

https://s3-ap-northeast-1.amazonaws.com/masui.org/b/6/b628468ad1775361aee7ff346b89bfa4.gif

Use menus and sliders in the realword

Scan the barcode and move

Video: FieldMouse

https://www.youtube.com/watch?v=ACj-TN2Upn0

MouseField

RFID reader + motion sensor

https://s3-ap-northeast-1.amazonaws.com/masui.org/d/9/d94214a01ebbac15c5714e13893e3f22.png

Using MouseField

Put something and move ⇒ search, control

https://s3-ap-northeast-1.amazonaws.com/masui.org/d/8/d8cd79ca279c4540b32c5ac25287f4ba.png

MouseField in the study

https://s3-ap-northeast-1.amazonaws.com/masui.org/3/0/3080e50da8a8b4003b7cabaa8322b84d.jpg

MouseField on a table

https://s3-ap-northeast-1.amazonaws.com/masui.org/d/5/d5f0a9bb2f393f2431b02d61313b46ef.jpg

PlayStand++

https://s3-ap-northeast-1.amazonaws.com/masui.org/6/1/61877d576aaa9751e4305da10f2db921.png

Using PlayStand++

https://s3-ap-northeast-1.amazonaws.com/masui.org/b/2/b2854fed63270061d12d976481d673bc.png

Video: PlayStand

https://www.youtube.com/watch?v=w-nEPbptLO8

GoldFish

Real-world GUI framework for Android

Read NFC ⇒ open browser ⇒ run JS ⇒ use sensors

All the behavior are described in JavaScript

Easily modifyable

Video: GoldFish

Real-world copy/paste

https://www.youtube.com/watch?v=1di7_2J8X0Y

GoldFish architecture

GoldFish client

Runs on Android

Sensor + display

GoldFish server

JavaScript program

URL redirect

GoldFish client

Automatic invocation by NFC "intent"

Display a Web page related to the NFC ID

Use sensor data on Android from JS program

http://gyazo.com/18b298f2b2a2851978504fcd46503938.png

GoldFish server

TinyURL, Bit.ly的な動作

Example 1: Showing a manual

http://gyazo.com/a2dcd1bfa983b4c8b81e406085e59c47.png

http://gyazo.com/de0fe7498b3a1fe856e07002f9a7ae1b.png

Examle 3: Rotation interface

http://gyazo.com/b06aee214b542952125f0d8a0ecb7abf.png

http://gyazo.com/f38d032e1c317e1bac30d52581b8317b.png

http://gyazo.com/ab301f469b9266c3f2d2ed07db3f7c22.png

http://gyazo.com/f00aef03e85ef6b1a838c527825a21fc.png

Video: Opening a door with Goldfish

http://www.youtube.com/watch?v=03sh3cXrXuw

Real-world copy/paste

http://gyazo.com/96ed656fe1f1007c299ec3817f4fe8d1.png

Goldfish examples

Controling appliances

Volume control

WiFi or IR

Control panel for appliances

Door authentication

Digital photoframe

Digital signage

...

Furniture computing

Put Goldfish in furniture

Control all appliances

Helpfeel

https://www.youtube.com/watch?time_continue=2&v=HqtIXi6Ot_M&feature=emb_logo

https://gyazo.com/91186d5bc22952546821938bff78c53a

"Vocabulary problem" in computer usage

Difference between query keyword and document text

"Set the clock" "Change the system time"

Solution by "ExpandHelp"

Generate all query expressions from regular expressions

Search query patterns

Don't search help entries

Regular expression (RE)

Specification of string patterns

Ambiguous search possible

a*bc

Got popular because of Unix tools

Correspond to state transition machine

Regular expression patterns

Selection

Character class

[a-z]

Repetition

Regular expression examples

abc

(abc|def)

[abc]

ab*c

(時計|時間|時刻)を([0-9]*)時に(セットする|設定する|あわせる)

Implementation of regular expression

Regular expression can be represented by state machine

A ← aS

B ← bA

C ← cA

B ← dC

http://gyazo.com/ef47894d05a71d085fb31cdd9527910e.png

Expanding a regular expression pattern

(時計|時間|時刻)を([0-9]*)時に(セットする|設定する|あわせる)

時計を0時にセットする

時計を0時に設定する

...

時間を1時にセットする

Useful for solving vocabulary problem

ExpandHelp

Compose help titles in regular expression and expand them

Search with approximate pattern matching

SFCHelp

Listing all questions of SFC

Generative Art

Generating art using computers

https://gyazo.com/ec1cd58c515466dd5d8d5edecf1964d5

https://gyazo.com/38a9479bec9280ce4a495259a62e79d9

https://gyazo.com/58af382c00c2b8a026acf1c0c1f86c8e

Painting by Paul Klee and its hommage

https://gyazo.com/c5012bf16088609c4163abf171d8fe7a

https://gyazo.com/0672eaade90cf9934a94e07aa1a5723a

https://gyazo.com/8d3c61f3e4902db97a703694a1c7658e

https://gyazo.com/c947ca3cd3fe997993cb4e0a623c4a6b

Fukatsu's system

http://fladdict.net/blog/2016/01/jackson-pollock.html https://gyazo.com/aa1e0cfbb288d42be431533e30a823ef.png

Book on Generative Art

https://digitalideation.github.io/ba_222_gencg_h1801/slides/week00.html#slide=1 https://gyazo.com/df252a34d637285c72478e100deb59f8

Evolution of computer interface

A computer I created using 8008 CPU

In 1976

http://Gyazo.com/078e8c514fec7baf6388b3bc6246423c.png

8008 computer

http://Gyazo.com/08817cf560e0b559b67dd0f7e121ec3f.png

Paper tape reader

http://Gyazo.com/3e8612d7427e1e95b93d87c0c71ae5f1.png

Alto

1973 @ PARC

First GUI workstation

Bitmap display

Mouse / window / menu

http://gyazo.com/96bf4fda6bf46b0f862f9d3c68905016.png

Steve Jobs and Alto

https://gyazo.com/d6273198fc881b8b35b1db4b61a1d005

Xerox Star

Introduced in 1981

https://gyazo.com/79d6aa012b0c679da9a11b02cb3f659e.png

JStar

Introduced in 1982

Japanese text handling

https://gyazo.com/966466ccdf1c07654c2c9574c8a0b459.png

Lisa

1983 @ Apple

GUI-based personal computer from Apple

Steve Jobs created this after seeing ALto

http://gyazo.com/c0596d8b63d9b9b790d3bb953c5291a7.png

Macintosh

1984 @ Apple

Originally a project by Jef Raskin

Steve Jobs ousted Raskin from the project

NeXT Computer

Founded in 1985 by Steve Jobs

Introduced NeXT Cube in 1988

Mac OSX

http://gyazo.com/66d94e93c6a0d0153f171b88692e2445.png

Evolution of development environment

Evolution of version control systems

SCCS (1972)

RCS (1980)

CVS (1986)

Subversion (2000)

git (2005)

Evolution of build systems

Make (1977)

Rake

ant

sbt

Various IDEs

Interface Builder (1986)

Installed on NeXTstep

Edit GUI objects visually

https://gyazo.com/26628fafb9d8cddd83ada076705b68fe.png

Xcode

https://gyazo.com/91768f4e2dd5e628bdbb9e09c26372cc.png

Eclipse

https://gyazo.com/6a4ba95d8e00c59eae210573099702d3.png

Evolution of development environment

35 years after Make

25 years after IDE

Almost the same for 20+ years

Why slow evolution?

People like same old way

People cannot adopt to new things

What is "intuitive"

People think something is intuitive when they know something well

e.g. pen operation, pulldown menu, ...

People can use them after long-time practice

Seven cafe by Kashiwa Sato

https://gyazo.com/182a24c8bc2b1322accaabcdfc0d8437.png

The failure of new design

https://gyazo.com/92be5ee1a15f21b8c9b557e0215fe775.png

The failure of new design

https://gyazo.com/b81ff77125b286c600bc609fd19b1421.png

New products do not become popular soon

Most people are very conservative